Manual control re-engagement in an autonomous vehicle

ABSTRACT

Vehicles may have the capability to navigate according to various levels of autonomous capabilities, the vehicle having a different set of autonomous competencies at each level. In certain situations, the vehicle may shift from one level of autonomous capability to another. The shift may require more or less driving responsibility from a human operator. Sensors inside the vehicle collect human operator parameters to determine an alertness level of the human operator. An alertness level is determined based on the human operator parameters and other data including historical data or human operator-specific data. Notifications are presented to the user based on the determined alertness level that are more or less intrusive based on the alertness level of the human operator and on the urgency of an impending change to autonomous capabilities. Notifications may be tailored to specific human operators based on human operator preference and historical performance.

BACKGROUND

Vehicles may be equipped with sensors and processing and communicationcapabilities that allow the vehicle to navigate autonomously withouthuman intervention. Autonomous vehicle navigation is not possible,however, under all circumstances. In some situations, a vehicle may lackthe capability to navigate autonomously such as adverse or extremeweather conditions, in the event of the loss of vehicle sensors or acommunications link, under disaster conditions, due to vehiclecollisions in the area, etc. When a vehicle can no longer navigateautonomously, control of the vehicle may be returned to a humanoperator. Transfer of control of the vehicle to a human represents apotential danger due to the vehicle's autonomous nature—a human operatorwho has been traveling in the vehicle may be inattentive to roadconditions and unprepared to make the decisions and exercise the motorcontrol needed to safely operate the vehicle.

SUMMARY OF THE DISCLOSURE

This summary is provided to introduce a selection of concepts in asimplified form that are further described in the Detailed Descriptions.This summary is not intended to identify key features or essentialfeatures of the claimed subject matter, nor is it intended to be used tolimit the scope of the claimed subject matter.

Systems and methods are disclosed for requesting engagement of manualcontrol in a vehicle. The vehicles may have the capability to navigateaccording to various levels of autonomous capabilities, where thevehicle has a different set of autonomous competencies at each level. Incertain situations, the vehicle may shift from one level of autonomouscapability to another. The shift may require more or less drivingresponsibility from a human operator. Sensors inside the vehicle maycollect human operator parameters to determine an alertness level of thehuman operator. An alertness level may be determined based on the humanoperator parameters and other data including historical data or humanoperator-specific data. Notifications may be presented to the user basedon the determined alertness level that are more or less intrusive basedon the alertness level of the human operator and on the urgency of animpending change to autonomous capabilities. Notifications may betailored to specific human operators based on human operator preferenceand historical performance.

In one aspect, a method of requesting engagement of manual control in avehicle is disclosed. The method may include identifying a humanoperator of the vehicle, the human operator being associated with adriving capability profile; determining human operator parametersassociated with the human operator of the vehicle; and determining analertness level of the human operator based on the human operatorparameters and the driving capability profile. The method may alsoinclude receiving an indication to transfer control of the vehicle tothe human operator; displaying a manual control request notification tothe human operator, the manual control request notificationcommunicating a request for manual control, the manual control requestnotification further being based on the determined alertness level; andtransferring control of the vehicle to the human operator if the humanoperator accepts the request for manual control.

In another aspect, a system for transferring control of a vehicle to ahuman operator is disclosed. The system may include an evaluatorconfigured to evaluate readiness of a human operator of a vehicle, theevaluator including one or more sensors; a vehicle security arbiterconfigured to determine a security threat to the vehicle; and anotification server configured to display a first manual control requestnotification to the human operator, the first manual control requestnotification being based at least in part on the readiness of the humanoperator and the security threat to the vehicle. The notification serverbeing further configured to display a second manual control requestnotification to the human operator, the second manual control requestnotification being different from the first manual control requestnotification; and the vehicle security arbiter being further configuredto transfer control of the vehicle to the human operator.

In yet another aspect, a method is disclosed for transitioning controlof a vehicle. The method may include navigating a vehicle according to afirst level of autonomous capability, the first level of autonomouscapability defining a first set of autonomous competencies of thevehicle; receiving a request to operate the vehicle according to asecond level of autonomous capability within a time period, the secondlevel of autonomous capability defining a second set of autonomouscompetencies, the second set of autonomous competencies being differentthan the first set of autonomous competencies; and notifying a humanoperator of the vehicle with a notification of an impending change invehicle capability to the second level of autonomous capability, thenotification depending at least in part on differences between the firstset of autonomous competencies and the second set of autonomouscompetencies. The method may also include navigating the vehicleaccording to the second level of autonomous capability.

The methods may be implemented via computer systems, and may includeadditional, less, or alternate actions or functionality. Systems orcomputer-readable media storing instructions for implementing all orpart of the method described above may also be provided in some aspects.Systems for implementing such methods may include one or more of thefollowing: a special-purpose computing device, a personal electronicdevice, a mobile device, a wearable device, a processing unit of avehicle, a remote server, one or more sensors, one or more communicationmodules configured to communicate wirelessly via radio links, radiofrequency links, and/or wireless communication channels, and/or one ormore program memories coupled to one or more processors of the personalelectronic device, processing unit of the vehicle, or remote server.Such program memories may store instructions to cause the one or moreprocessors to implement part or all of the method described above.Additional or alternative features described herein below may beincluded in some aspects.

This summary is provided to introduce a selection of concepts in asimplified form that are further described in the Detailed Descriptions.This summary is not intended to identify key features or essentialfeatures of the claimed subject matter, nor is it intended to be used tolimit the scope of the claimed subject matter.

Advantages will become more apparent to those of ordinary skill in theart from the following description of the preferred aspects, which havebeen shown and described by way of illustration. As will be realized,the present aspects may be capable of other and different aspects, andtheir details are capable of modification in various respects.Accordingly, the drawings and description are to be regarded asillustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an example system for manual controlre-engagement in a vehicle.

FIG. 2 is a time-series schematic diagram of an example system formanual control re-engagement in a vehicle with a human occupant.

FIG. 3 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 4 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 5 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 6 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 7 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 8 is a schematic diagram of a vehicle equipped with an examplesystem for manual control re-engagement in a vehicle approaching a roadhazard.

FIG. 9 is an in-vehicle view of an example system for manual controlre-engagement in a vehicle.

FIG. 10 illustrates example operations for requesting engagement ofmanual control in a vehicle.

FIG. 11 illustrates example operations for transitioning control of avehicle.

DETAILED DESCRIPTIONS

Autonomous vehicles may exercise a range of capabilities when navigatingon open road conditions. An autonomous vehicle need not be viewed asoperating purely autonomously or purely manually. The Society ofAutomotive Engineers (SAE) has identified at least six levels ofautonomous vehicle capability ranging from no driving automation (Level0) to full automation (Level 5). As a vehicle moves up the levels ofcapability, additional autonomous competencies are added to thevehicle's set of skills (e.g., adaptive cruise control, parking assist,lane assist, traffic jam assist, conditional automation, highautomation, full automation, etc.).

At the various points on the vehicle's autonomous skill ladder, a humanoperator exercises an appropriate level of manual control. If a vehiclesupplies only adaptive cruise control or parking assist capabilities,then the human operator must exercise a high level of manual control andis responsible for any and all non-autonomous aspects of the vehicle. Ifa vehicle supplies high or full automation, on the other hand, a humanoperator may participate at a low level, or even not at all, innavigating the vehicle. If a vehicle exercises a high level ofautonomous capability (e.g., Levels 4 or 5), a human operator may becomedisengaged from the road and unaware of the road environment surroundingthe vehicle. The human user may become focused on non-driving tasks(e.g., reading, working, playing games, conversations with otherpassengers, phone calls, etc.). A human operator may even go to sleepand lose all contact with controlling the vehicle.

It may be desirable for an autonomous vehicle to shift from one level ofautonomous capability to another, such as due to changing roadconditions, weather conditions, due to a vehicle crash, disaster oremergency situation, etc. To the extent the human operator will need tochange her involvement in piloting the vehicle when the vehicle shiftsbetween levels of automation, she must be notified of the impendingchange. A variety of notification types are used in the system formanual control re-engagement to communicate with the human operatorregarding upcoming changes in the human operator's responsibilities inpiloting the vehicle. A notification may be more intrusive to the humanoperator or less intrusive to the human operator depending on theurgency of the impending change to the vehicle's capabilities and thehuman operator's responsibilities.

Different human operators will respond differently to notifications ofchanges to the vehicle's autonomous capabilities. Human operatorsexhibit differences from one another in terms of attention span, abilityto multitask, ability to shift focus from one activity to another afterreceiving a notification, etc. To some extent, these differences arenatural characteristics of the human operators themselves (e.g., somepeople are more likely than others to become engrossed in reading a bookand may take more time to respond to a notification of impending vehicleautonomous capability change than other people who may tend not tobecome engrossed in reading a book while riding in a car). In othersituations, a human operator's ability to acknowledge a notification andprepare to exercise more or less control over the vehicle may depend onother aspects of the human operator that may change over time (e.g.,intoxication level, how well rested the human operator is, whether thehuman operator has become focused on another activity or remains awareof the vehicle's surroundings, the health of the human operator, etc.).In yet other situations, a human operator's ability to acknowledge anotification and prepare to exercise more or less control over thevehicle may change slowly over time (e.g., a human operator's eyesightmay deteriorate over time, motor control deteriorates with age, etc.).

A change to a vehicle's autonomous capability may also have a variabletime target in which the change should be made. Some changes to avehicle's autonomous capabilities must be made quickly, such as inemergency situations (e.g., a vehicle crash or other rapidly approachingroad hazard, if the human operator experiences an emergent medical orhealth problem, etc.). Other changes to a vehicle's autonomouscapabilities need not be made quickly (e.g., if adverse or extremeweather conditions are forecasted but not yet experienced, if a humanoperator is nearing the limits of her ability to stay awake, if a humanoperator experiences abnormal but not threatening health issues, etc.).

A vehicle may receive an indication from third-party sources of animpending road hazard and may determine the urgency of altering thevehicle's autonomous capabilities on its own. Vehicles may communicatewirelessly with one another in to relay updates to one another regardingchanging road conditions as experienced by the vehicles them selves. Thevehicles may communicate according to a peer-to-peer network in whichthe vehicles connect with one another directly (e.g., a swarm) or acentralized party may collect information from the vehicles, optionallyprocess the information, and selectively provide relevant information tovehicles as the vehicles need it (e.g., client/server). For example, ifa vehicle crash occurs on a road and vehicles in the vicinity detect thecrash occurrence, the vehicle in the vicinity of the crash maycommunicate an emergency signal to vehicles approaching the crash siteon the road such that those vehicles may take precautions to avoidcrashing into any vehicles that are slow, stopped, or otherwisenavigating differently than expected due to the crash.

In another implementation, a vehicle may receive an indication from athird-party that the vehicle should alter its autonomous capabilitiesand may receive a target time period in which to make the changetherewith. A centralized authority (e.g., a weather prediction bureau,an insurer, a vehicle owner, a government agency, etc.) may determinethat a change should be made to a vehicle's autonomous capabilitiesbased on information regarding the vehicle's environment and maycommunicate a request to make the change to the vehicle. Such a requestmay accompany a target time in which to make the change to the vehicle'scapabilities.

A vehicle may adjust its autonomous capabilities to increase or decreasethe number of autonomous capabilities, depending on the situation thevehicle is in. In some situations, the vehicle maybe ill-suited tosafely navigate a situation autonomously, and manual control may bepreferable (e.g., navigation in close quarters with other vehicleswherein human communication is needed to coordinate with the operatorsof other vehicles, if the vehicle experiences a sensor or other hardwarefailure, if the vehicle loses a communications link, etc.). In othersituations, a human operator may be more ill-suited to safely navigate asituation than the vehicle would be autonomously. It may be known thathuman operators in general, or a specific human operator, is likely tomake a mistake that could lead to a crash that an autonomouslycontrolled vehicle would not make. For example, if a vehicle istraveling at a high rate of speed in low visibility conditions (e.g.,heavy fog, frequent road elevation changes, blind spots, etc.) and isapproaching a sudden traffic jam wherein other vehicles are traveling ata comparatively much slower rate of speed or are stopped on a road, itmay be known that human operators are less likely to reduce speed intime than an autonomous vehicle would be. If the vehicle detects such aroad condition approaching, it may request the human operator relinquishsome or all control of the vehicle to reduce risk of a vehicle crash.

FIG. 1 is a schematic diagram of an example system 100 for detectinghuman operator parameters in a vehicle. The system 100 includes avehicle 102 with more than one level of autonomous capabilitiesnavigating on a road 104. The vehicle 102 may include one level of puremanual control and one or more additional levels of enhanced automatedcontrol. When the vehicle approaches a potential road hazard 106 orother road conditions under which a change in the autonomouscapabilities of the vehicle 102 are warranted, the vehicle 102 maydisplay a notification to the human operator 110 to expect an upcomingor immediate change in the responsibilities of the human operator 110 inpiloting the vehicle 102.

One factor in determining the type of notification of a change inautonomous capabilities of the vehicle 102 to the human operator 110 isthe current state of the human operator 110. An alertness level of thehuman operator 110 may be estimated based on objective data collected bysensors inside the vehicle 102 such as the sensors shown in the bubble108 of the interior of the vehicle 102. In one implementation, anoptical imaging device 112 (e.g., a camera) is located inside thevehicle 102 and is directed towards the human operator 110. The camera112 may capture images of the human operator 110 that may be analyzed toextract human operator parameters used to determine an alertness stateof the human operator.

In at least one implementation, the human operator parameters include abody position of the human operator 110. If the human operator 110 issitting upright in the seat 114 and has her hands on or near a steeringwheel, it is likely that the human operator 110 will be more responsiveto a change in driving responsibilities than if the human operator 110is reclined in the seat 114 without hands near the steering wheel.Images captured by the camera 112 may be analyzed by components of thevehicle 102 to determine whether the human operator is in a bodyposition that indicates a greater or lower level of alertness. Thecamera 112 may capture a series of images of the human operator 110(e.g., a video) that may be compared by components of the vehicle 102 todetermine an activity level of the human operator 110. For instance, ifthe human operator 110 is asleep, then she will likely exhibit adifferent movement patterns than a human operator 110 who is awake.Another type of objective data that may be collected by the camera 112regarding the human operator's alertness and preparedness for acceptinga change in driving responsibilities is the activity in which the driveris engaged. If analysis of images captured by the camera 112 indicatethat the human operator 110 is holding a book or electronic device, forexample, then the human operator 110 is more likely to experience aslower change of focus away from the book and to road conditions than ahuman operator 110 who is not holding a book or electronic device. Thecamera 112 may also capture images of the face of the human operator 110to determine whether her eyes are open or closed, focused on theenvironment outside of the vehicle 102 or inside the vehicle 102 andother factors such as fatigue.

The human operator parameters may also include non-visual data collectedfrom the interior of the vehicle 102. In at least one implementation,non-visual data includes biometric data of the human operator (e.g.,heart rate, breathing rate, body temperature, perspiration rate, etc.).Biometric data may be collected via the seat 114 in the vehicle 102because the seat is in physical contact with the human operator 110,which facilitates the collection of various types of biometric data. Forexample, a sensor 116 may be embedded in the seat such that the sensor116 can collect relevant data (e.g., a thermometer, a heart rate sensor,a breathing rate sensor, a perspiration sensor, etc.).

Other types of sensors that not embedded in the seat 114 may also beused to collect non-visual data for the human operator parameters. Inone implementation, an audio sensor 118 detects sounds inside thevehicle 102. If a human operator is speaking to another occupant of thevehicle 102 or on a phone, the human operator's speech will berecognized by the audio sensor 118. The human operator parameters maythen reflect that the human operator 110 is awake and likely somewhatbut not completely focused on the environment surrounding the vehicle102. The sensor 118 may also detect other types of sounds such as thehuman operator 110 moving inside the vehicle 102, suggesting an activitylevel of the human operator 110.

Human operator parameters may also include historical informationregarding the human operator 110. In one implementation, components ofthe vehicle 102 compile historical data regarding the human operator110. For example, the speed at which the human operator 110 responded tovarious notifications of an upcoming change to the vehicle's autonomouscapabilities (e.g., notifications of various levels of intrusiveness,audio notifications, video notifications, haptic feedback notifications,etc.) As another example, the vehicle 102 may compile, how well thehuman operator 110 reacted to various types of road conditions andpotential hazards (e.g., how often does the human operator 110 engage inemergency braking in various road conditions such as snow or rain, doesthe human operator 110 tend to lane drift at high speed, is the humanoperator 110 prone to panicked control movements in heavy traffic,etc.).

Historical information regarding the human operator 110 may also bereceived via a communications device at the vehicle 102 from athird-party. In one implementation, an insurer of the vehicle 110compiles historical driving information regarding the human operator 110such as vehicle crash history, vehicle usage history, and biographicaldata such as age and education level that may be used to determine analertness level of the human operator 110. For example, if the insurerknows that the human operator 110 usually operates the vehicle early inthe morning and never at night, the insurer may determine that thathuman operator 110 is not accustomed to driving in the middle of thenight. If the human operator 110 does drive one day in the middle of thenight, then the insurer may determine that this poses a greater thanusual safety risk for the particular human operator 110. Other rules mayalso be applied based on historical data such as young drivers under age18 pose a greater risk when driving late at night, in adverse weatherconditions, when the audio sensor 118 detects multiple young people inthe vehicle 102 at the same time, etc.

FIG. 2 is a time-series schematic diagram of an example system 200 formanual control re-engagement in an autonomous vehicle 206 with a humanoperator 202. The vehicle 206 includes an image capture device 204 suchas a camera. The camera 204 may be directed towards the face of thehuman operator 202 and images captured thereof. Components of thevehicle 206 may analyze the images of the face of the human operator 202to identify characteristics of the human operator 202 that are relevantto the human operator parameters used to determine an alertness level ofthe human operator 202.

The camera 204 may analyze various features of the human operator 202 tosupply human operator parameters to the vehicle 206. For example, in atime period before time 210, the eyes of the human operator 202 are openand focused outside the window of the vehicle 206. The camera 204 maycapture a series of images of the face of the human operator 202 todetect other features such as rapidity of eye movement, dilation of eyepupils, blinking, etc.

In a time period after time 210, the camera 204 may capture more imagesof the face of the human operator 202. In a time period after timeperiod 210, the eyes of the human operator 202 are still open, but arefocused on a book 214. Images captured by the camera 204 may reflectthat the eyes of the human operator 202 are no longer focused on theenvironment surrounding the vehicle 206, but instead are focused on thebook 214. Images captured by the camera 204 may record features of theeyes of the human operator 202 such as eye movements indicating a speedat which the human operator 202 is reading. Slower reading speeds andother eye movements may indicate a fatigue level of the human operator202.

If the eyes of the human operator 202 are no longer focused on theenvironment surrounding the vehicle, human operator parameters mayinclude a level of distraction represented by a “score.” For example, ifthe human operator 202 is focused on a book 214, it may be likely thatthe human operator is engaging in an extended period of perhaps intenseconcentration on the book 214. The longer the human operator 202 focuseson the book 214, the more likely she is to have a higher level ofdetachment from her environment. Such behavior may indicate that thehuman operator parameters should reflect a higher level of distraction.A security arbiter in the vehicle 206 may set a relatively longer periodof time that would be expected before the human operator 202 responds toa notification of change to driving responsibility based on a higherdistraction score in the human operator parameters. On the other hand,if the human operator 202 is only occasionally focusing on a handhelddevice (e.g., checking email, etc.), then the distraction of the humanoperator 202, while still present, may not be considered as distractedas long periods of reading a book 214. In such a case, a lowerenvironmental detachment score may be included in the human inputparameters and relied on by other components of the vehicle 206 to alertthe human operator of an impending change in driving responsibility andmanagement of shifting the vehicle from one level of autonomous controlto another.

At a time period after time 212, the camera 204 may capture additionalimages of the face of the human operator 202 that indicate the humanoperator 202 is suffering from fatigue. Images captured by the camera204 may show that that eye lids of the human operator 202 are not asopen as before the human operator 202 began experiencing fatigue. Othereye-based indications include blinking rate, location of eye focus, andeye movement rapidity.

The human operator parameters collected by the camera 204 are used todetermine an alertness level of the human operator 202. The determinedalertness level may be used to determine whether a change should be madeto the autonomous capability level of the 206 and whether the humanoperator 202 should take on more or less driving responsibility. If thevehicle 206 determines that an approaching road hazard 208 exists andthe human operator 202 should take on more driving responsibility, thenthe vehicle may display a notification to the human operator 202. Thetype of notification displayed to the human operator 202 and the timeperiod for which the notification should be displayed in advance of anencounter with the road hazard 208 depend on the alertness level of thehuman operator 202 and the human operator parameters sensed bycomponents of the vehicle 206. For example, if the human operator 202has a higher alertness level, such as in the time period before time210, then the notification of a change to the vehicle's autonomouscapabilities may be less intrusive or occur closer to the road hazard208 because it is expected that the human operator 202 will be able torecognize the notification and increase driving responsibilityrelatively quickly. On the other hand, if an alertness of the humanoperator 202 is lower due to human operator parameters such as thoseexamples after time 210 and after time 212 (e.g., occupied with anothertask, experiencing fatigue, etc.), then the notification of a change tothe vehicle's autonomous capabilities may be more intrusive or occurfarther away from the road hazard 208 because it is expected that thehuman operator 202 will need relatively more time to recognize thenotification and prepare to increase driving responsibility.

FIG. 3 is an in-vehicle view of an example system for manual controlre-engagement 300 in an autonomous vehicle. Included in the system 300are a number of types of notifications to alert a human operator thatthe vehicle may shift to a different level of autonomous capability. Thenotifications range from less intrusive to a human operator to moreintrusive to a human operator, depending on the urgency of the shift toanother autonomous vehicle capability level and/or any approaching roadhazards. Notifications that the vehicle will shift to a different levelof autonomous capability may be tailored to an individual human operatorbased on collected human operator parameters, such as whether theparticular human operator is in an occupied or distracted state, whetherthe particular human operator tends to respond well to certain types ofnotifications over other types of notifications, whether the humanoperator exhibits biomarkers that indicate a reduced capacity fordriving (e.g., accelerated or irregular heart rate), whether animpending shift will require a greater or lesser degree of vehiclecontrol on behalf of the human operator, etc.

In the implementation illustrated in FIG. 3, a text notification 302appears on the dashboard of the vehicle. In one implementation, the textnotification 302 alerts a human operator of an impending change to thelevel of autonomous control exercised by the vehicle. For example, ifthe vehicle or another entity has determined that a greater degree ofautonomous control is better suited to the potential road hazard, thenthe text notification 302 may indicate to the human operator that thevehicle will include greater autonomous capability (and thus less humanresponsibility) to navigate the upcoming road hazard. The textnotification may include information regarding the road hazard or animpending shift of autonomous vehicle capabilities (e.g., whether thehuman operator is expected to exercise a greater or lesser degree ofcontrol, the urgency of the hazard or shift in capabilities, an expectedtime until the hazard is encountered or the shift in capabilities isexecuted, etc.).

The text notification 302 may be adjusted to be more intrusive or lessintrusive to the human operator, depending on the nature of the roadhazard or change in autonomous capabilities. If the vehicle is preparingto gain more control over navigation of the vehicle with a correspondingreduction in responsibility on the part of the human operator, then thenotification 302 need not be highly intrusive because the notification302 is more of an informational nature. On the other hand, if there is aproblem with the vehicle's autonomous capabilities and the humanoperator is expected to take some or all responsibility for piloting thevehicle, then the notification 302 may be more intrusive to the humanoperator because the human operator must be made aware of the alert,which could involve interrupting another activity that the humanoperator may be engaged in (e.g., sleeping, reading, working, inconversation with another occupant of the vehicle, etc.). The textnotification 302 may be made more or less intrusive to the humanoperator by changing text font, flashing text, increasing or decreasingbrightness of the text notification 302, using colors to indicateurgency (e.g., red for high urgency, yellow or intermediate urgency,green for lower urgency, etc.)

In the implementation illustrated in FIG. 3, haptic feedback 304 may beused to notify a human operator of an impending road hazard or change inautonomous vehicle capabilities. Haptic feedback 304 may emanate fromhaptic devices embedded in various parts of the interior of the vehiclesuch as in the vehicle seats, arm rests, headrests, vehicle and/or mediacontrol surfaces, etc. Haptic feedback signals 304 may accompany a textnotification 302. Haptic feedback signals 304 may be used to make anotification more intrusive to a human operator. For example, if a humanoperator is deemed to be in a state of lowered attention span, such aswhen sleeping, reading a book, etc., then a haptic feedback signal 304may assist in helping the human operator to “snap out” of her currentstate and to re-focus on the notification 302 and/or any road or vehicleconditions that require her attention.

Additionally, or alternatively, an audio alert 306 may be used to notifya human operator of an impending road hazard or impending change inautonomous vehicle capabilities. An audio alert 306 may emanate fromspeakers located around the interior of the vehicle. Audio alert 306 mayinclude sounds designed to attract the attention of a human operatorand/or voice commands directed to the human operator. For example, avoice command in the audio alert 306 may instruct the human operator toprepare for an increase in driving responsibility or signal to the humanoperator how much time is left until the human operator will be expectedto exercise more or less control over the vehicle. The audio alert 306may be made more intrusive or less intrusive to a human operator byvarying the volume of the audio alert 306, including a sound reflectiveof the urgency of impending changes to the human operator'sresponsibilities (e.g., an alarm sound for an urgent hazard, a softvoice or bell for a less urgent hazard, etc.).

In at least one implementation, other visual alerts 308 are included tonotify a human operator of an impending road hazard or impending changein autonomous vehicle capabilities. In one implementation, the visualalerts 308 include a light bulb (e.g., a light emitting diode) forflashing or steady illumination. In other implementations, the visualalerts 308 include an e-fabric display of images or colors on theinterior of the vehicle (e.g., an alert symbol, a text message, a colorindicating urgency, etc.). Like other alerting mechanisms in thevehicle, the visual alert 308 may be adjusted to be more intrusive orless intrusive to the human operation by varying the intensity of thealert (e.g., brightness of a light bulb), flashing the visual alert 308,changing the color of the visual alert 308, changing a message or symboldisplayed by the visual alert 308, etc.

FIG. 4 is an in-vehicle view of an example system for manual controlre-engagement 400 in a vehicle. The system 400 includes a text display402 for alerting a human operator of an impending change to autonomousvehicle capabilities of the vehicle. In the implementation illustratedin FIG. 4, the vehicle or another party has determined that an expectedroad hazard should not be navigated by the vehicle autonomously. Someroad hazards (e.g., a rock slide) are very serious and seldomencountered by autonomous vehicle. It may not be known how well anautonomous vehicle is able to navigate such a hazard due tounpredictable conditions and infrequent collection of the behavior ofautonomous vehicle in such a situation. In such cases, it may bedetermined that a human operator must quickly assume control of thevehicle.

The determination that an expected road hazard should not be navigatedautonomously may be based on information collected by the vehicle itselfor may be reported to the vehicle via a communications interface (e.g.,other vehicles in the area may send reports that they are notsuccessfully navigating a road hazard, an insurer may determine thathuman operators in general or a human operator in particular is morelikely to safely navigate a road hazard than a vehicle under autonomouscontrol, etc.). The text notification 402 may be accompanied by othernotification methods (e.g., audio signal, haptic feedback, visualsignals, etc.). The text notification may further include informationsuch as an expected amount of time until a road hazard will be reachedor an expected amount of time until a change to autonomous drivingcapabilities takes effect.

FIG. 5 is an in-vehicle view of an example system for manual controlre-engagement 500 in a vehicle. The system 500 includes a textnotification 502. Like the text notifications disclosed herein withreference to other implementations, the text notification 502 includes amessage to the human operator regarding an impending road hazard and/oran impending change to the autonomous capabilities of the vehicle. Textnotification 502 may include a disruption to other content displayedinside the vehicle such as presentation of video, entertainmentinformation (e.g., video games, web surfing, books, etc.), maps, etc.Interruption of other content may make the notification 502 moreintrusive into the human operator's environment, which may beappropriate depending on the severity of the alert to be displayed.

The system 500 may include a warning alert 504 that may be displayed atvarious points around the interior of the vehicle. In at least oneimplementation, a warning alert 504 is displayed in locations wherein itis likely to attract the attention of a human operator. For example, thewarning alert 504 may be displayed on a rear-view mirror, on thesteering wheel, in a heads-up display on the windshield, etc. Whether awarning alert 504 is to be displayed depends in part on an urgency ofattracting the human operator's attention based on the human operatorparameters and alertness level of the human operator.

FIG. 6 is an in-vehicle view of an example system for manual controlre-engagement 600 in a vehicle. In the implementation illustrated inFIG. 6, a notification 602 includes a request to the human operator toadjust the level of autonomous control exercised by the vehicle. In somesituations, the vehicle may determine or a third party may determine,that a vehicle is likely to be navigated more safely autonomously thanby the human operator. Examples of third party entities that may makesuch a determination include a vehicle manufacturer, a governmentagency, an insurer, a vehicle owner, etc. The vehicle or the thirdparties may also determine that a vehicle is likely to be operated moresafely manually than autonomously in certain situations. In the exampleillustrated in FIG. 6, an extreme weather alert issued from a weatherservice initiates a request to the human operator to agree to allow thevehicle reduce or eliminate autonomous capabilities in favor of manualcontrol. A human operator may interact with the notification 602directly or by other controls in the interior of the vehicle to acceptthe request presented in notification 602.

FIG. 7 is an in-vehicle view of an example system for manual controlre-engagement 700 in an autonomous vehicle. In at least oneimplementation, an insurer determines whether a human operator or avehicle is more likely to safely navigate the vehicle in certainconditions. The insurer initiates a request to the human operator to bedisplayed as notification 702. The notification 702 may include an offerfrom an insurer to lower insurance premium prices in return for thehuman operator's agreement to allow the vehicle to assume a greater orlesser degree of autonomous control over road navigation depending onthe rules preferred by the insurer. Rules preferred by the insurer maydepend on factors such as the human operator parameters measured insidethe vehicle, a driving history of the particular human operator in thevehicle, and data gathered regarding events outside the vehicle.

FIG. 8 is a schematic diagram 800 of a vehicle 802 equipped with anexample system for manual control re-engagement in an autonomous vehicleapproaching a road hazard 804 on a road 806. Under some circumstances,it may be safer for the vehicle 802 to alter available autonomouscapabilities to more safely navigate the road hazard 804. To make adetermination whether it is safer to alter autonomous vehiclecapabilities or to remain in the status quo, the vehicle detects avariety of human operator parameters regarding a human operator of thevehicle to determine a human operator alertness level.

Also relevant to a determination of whether it is safer to alterautonomous vehicle capabilities of the vehicle 802 is an assessment ofconditions on the road 806 and of the road hazards 804. In at least oneimplementation, the vehicle 802 receives information regardingconditions on the road 806 and of the road hazard 804 from othervehicles and from sensors in the road 806 itself.

The vehicle 802 includes components for managing a transition from onelevel of autonomous capabilities to another level of autonomouscapabilities. One type of component to manage the transition is anevaluator in the vehicle 802 to evaluator to evaluate readiness of ahuman operator of a vehicle including one or more sensors. A securityarbiter in the vehicle 802 determines a security risk to the vehicle(e.g., a road hazard, adverse conditions, diminished capacity of a humanoperator, etc.) and determines whether a level of autonomous capabilityof the vehicle should be adjusted by adding or removing autonomouscapabilities and conversely adding or removing driving responsibilityfrom a human operator. The security arbiter in the vehicle 802 mayreceive human input parameters directly from the human operator and/orfrom a remote party.

In at least one implementation, the security arbiter in the vehicle 802evaluates a threat posed by the road hazard 804. The security arbitermay receive information regarding the threat from other roadparticipants and remote parties. For example, vehicles 808 traveling inthe same direction as the vehicle 802 but closer to the road hazard 804may wirelessly relay information back to vehicle 802 and the othervehicles approaching the road hazard 804 on the road 806 behind thevehicles 808. The security arbiter may receive indications, for examplewithout limitation that the vehicles 808 have encountered sharply lowerroad speeds or have observed adverse road conditions. The vehicles 808may further relay information to the vehicle 802 regarding the locationof the road hazards 804 on the road 806. The vehicles 808 may includetelematics data in the information sent to the security arbiter in thevehicle 802 (e.g., heavy braking, high G-forces, etc.) As anotherexample, third parties (e.g., government agencies, insurers, vehicleowners, etc.) may collect information regarding the conditions on theroad 806 by way of remote sensors 812. The remote sensors 812 may detectroad speeds and be able to determine whether vehicles are in distress orif a dangerous condition has developed on the road 806. Remote sensors812 may be fixed in place roadside (e.g., vibration sensor, vehiclecounter, camera, etc.) or may be mobile sensors (e.g., drone, unmannedaerial vehicle, helicopter, etc.).

After receiving data regarding the road hazard 804, the security arbitermay determine a security threat to the vehicle based on the road hazard804 and other information available to the security arbiter. Thesecurity arbiter may further determine a change to autonomous vehiclecapabilities that should be taken to improve safety when navigating theroad hazard 804. Such a determination may be based on informationavailable to the security arbiter or it may be an instruction receivedfrom a third party. The security arbiter may further determine a timeperiod during which the change in the vehicle's autonomous capabilityshould occur.

Another component in the vehicle 802 is a notification server. Thenotification server in the vehicle 802 may include hardware and softwarecomponents (e.g., a display for text messages to a human operator insidethe vehicle, speakers for playing audio text notifications andinstructions, lights, feedback devices, an operating system,microphones, etc.) for presenting information to and receivinginformation from a human operator of the vehicle 802. The notificationserver presents notifications to a human operator of the vehicle 802. Inat least one implementation, the security arbiter determines a timeperiod and an intrusiveness level of

After receiving data regarding the road hazard 804, the security arbitermay select a time period during which the vehicle should transition to adifferent level of autonomous capability. The time period may becalibrated based on the speed of the vehicle 802 on the road 806, thelocation of the road hazards 804, information received from vehicles808, etc. In at least one implementation, the security arbiter maydecrease speed of the vehicle to lengthen the time until the vehicle 802reaches the road hazard 804. In at least one implementation, thesecurity arbiter provides a time period to the notification server ofthe vehicle 802 during which changes to the vehicle's autonomouscapabilities should be made. In implementations, a security arbiterrequests the notification server to require human operatoracknowledgement before implementing changes to the autonomouscapabilities of the vehicle 802. In another implementation, the securityarbiter will cease navigation of the vehicle if the human operator doesnot acknowledge increased driving responsibilities. In otherimplementations, the security arbiter implements a change in theautonomous capabilities of the vehicle 802.

FIG. 9 is an in-vehicle view of an example system 900 for manual controlre-engagement in a vehicle. The system 900 includes a text notification902 including request for acknowledgement from the human operator. Thetext notification 902 may serve as a “ping” to the human operator totest for responsiveness. When a human operator responds to the textnotification 902 (e.g., through a touch interface, pushing a button,speaking a response into a microphone, etc.), the system 900 (e.g., asecurity arbiter on the vehicle) may record a responsiveness timeassociated with the human operator's acknowledgement of the textnotification 902. An intrusiveness level of the text notification 902may be varied to test the human operator's reaction to different levelsof intrusiveness. Some human operators may react well to minimallyintrusive notifications and find more intrusive notifications to beirritating. Other human operators, on the other hand, may not respondwell to minimally intrusive notifications and may need more intrusivenotifications to respond to a request by the vehicle to assume moredriving responsibility.

FIG. 10 illustrates example operations 1000 for requesting engagement ofmanual control in a vehicle. An identifying operation 1002 identifies ahuman operator of a vehicle, the human operator being associated with adriving capability profile. The identifying operation 1002 may rely onsensors inside the vehicle to identify the human operator (e.g., facialrecognition) or the identifying operation may identify the humanoperator of the vehicle via credentials (e.g., bio credentials,answering a cryptographic challenge, username and password, possessionof a physical vehicle key, etc.). The driving capability profile may beformed from data collected regarding the human operator. The dataregarding the human operator may include driving history data such asprevious reactions to road hazards, previous vehicle crashes, dataregarding the type of notifications the human operator tends to respondwell to, the amount of time the human operator typically needs torespond to a notification, the amount of time the human operatortypically needs to acknowledge an acknowledgement request, types of roadconditions under which the human operator typically exhibits good orpoor driving skills, the frequency with which the human operator becomesintoxicated or otherwise experiences diminished driving capabilities,the human operator's schedule, whether the human operator is likely tobe sleep-deprived, etc.

A determining operation 1004 determines human operator parametersassociated with the human operator. The determining operation 1004 mayinclude sensors inside the vehicle to collect biometric data regardingthe human operator (e.g., camera, heart rate sensor, perspirationsensor, steering wheel grip force sensor, body temperature sensor,movement sensors, microphones, etc.). The determining operation 1004 mayalso rely on information about the human operator stored by the vehicleor communicated to the vehicle such as demographic information regardingthe human operator (age, gender, length of driving experience, etc.) andhistorical information regarding the human operator's usage of thevehicle.

Another determining operation 1006 determines an alertness level of thehuman operator based on the human operator parameters and the drivingcapability profile. The determining operation 1006 may include adetermination of an amount of time the human operator is likely to needto respond to notifications of various levels of intrusiveness.

A displaying operation 1008 displays a manual control requestnotification to the human operator. The manual control requestnotification may be based on the determined alertness level of the humanoperator. If the determining operation 1006 determines that a humanoperator has a low alertness level, then the displaying operation 1008may display a more intrusive notification to the human operator and maydisplay the notification earlier than if the human operator has a highalertness level. If the determining operation 1007 determines that ahuman operator has a high alertness level and the driving capabilityprofile and/or the human operator parameters indicate that the humanoperator responds well to a certain type of notification, then thedisplaying operation 1008 may display that notification to the humanoperator.

A transferring operation 1010 transfers control of the vehicle to thehuman operator if the human operator accepts the request for manualcontrol displayed in operation 1008. In one implementation, a humanoperator accepts the request for manual control by interfacing with aninput such as a touch interface or control inside the vehicle. Inanother implementation, the human operator accepts the request formanual control by moving the vehicle controls (e.g., steering wheel,brake pedal, etc.).

FIG. 11 illustrates example operations 1100 for transitioning control ofa vehicle. A navigating step 1102 navigates a vehicle according to afirst level of autonomous capability. The first level of autonomouscapability includes a set of autonomous competencies exercised by thevehicle while it is operating according to the first level of autonomouscapability. Autonomous capabilities include without limitation adaptivecruise control, parking assist (steering only), lane assist, fullparking assist, traffic jam assist, conditional automation, highautomation, and full automation. These autonomous capabilities may beorganized according to ascending levels (e.g., Levels 0-5) as theautonomous competencies take over more and more driving responsibilitiesfrom a human operator. A vehicle navigating according to a first levelof autonomous capability exercise some (or all) of these autonomouscompetencies, depending on which autonomous competencies are included inthe first level. For example, if the first level of autonomouscapability is Level 3 autonomous capability, then a vehicle navigatingaccording to the first level of autonomous capability may exercise laneassist and parking assist, but not conditional automation.

A receiving operation 1104 receives a request to operate the vehicleaccording to a second level of autonomous capability. The receivingoperation 1104 may receive the request from an entity separate from thevehicle itself. For example without limitation, a government agencymonitoring road conditions, a vehicle manufacturer, an insurer, avehicle owner, etc. may send a request to the vehicle to operatoraccording to a second level of autonomous capability. In anotherimplementation, the vehicle itself may determine that navigation islikely to be safer if the vehicle shifts to a different level ofautonomous capability. For example, a security arbiter on the vehiclemay evaluate human operator parameters and a human operator alertnesslevel in comparison to any known or expected road hazard to create therequest to operate the vehicle according to the second level ofautonomous capability, and the vehicle may perform the receivingoperation 1104 in response to the security arbiter's request.

A notifying operation 1106 notifies a human operator with a notificationof an impending change to the vehicle's autonomous capabilities. In oneimplementation, the notifying operation 1106 is performed by anotification server on the vehicle. The notification may be adjusted bythe notification server to be more intrusive or less intrusive to thehuman operator depending on the information conveyed therein. Forexample, if the impending change in vehicle capability includes areduction to the driving responsibilities of the human operator, a lessintrusive notification may be selected. On the other hand, if theimpending change in vehicle capability includes an increase in drivingresponsibility of the human operator, then a more intrusive notificationmay be used. A navigating step 1108 navigates the vehicle according tothe second level of autonomous capability.

Of course, the applications and benefits of the systems, methods andtechniques described herein are not limited to only the above examples.Many other applications and benefits are possible by using the systems,methods and techniques described herein.

Furthermore, when implemented, any of the methods and techniquesdescribed herein or portions thereof may be performed by executingsoftware stored in one or more non-transitory, tangible, computerreadable storage media or memories such as magnetic disks, laser disks,optical discs, semiconductor memories, biological memories, other memorydevices, or other storage media, in a RAM or ROM of a computer orprocessor, etc.

The patent claims at the end of this patent application are not intendedto be construed under 35 U.S.C. § 112(f) unless traditionalmeans-plus-function language is expressly recited, such as “means for”or “step for” language being explicitly recited in the claim(s). Thesystems and methods described herein are directed to an improvement tocomputer functionality, and improve the functioning of conventionalcomputers.

What is claimed:
 1. A method of requesting engagement of manual controlin a vehicle, the method comprising: identifying a human operator of thevehicle, the human operator being associated with a driving capabilityprofile; determine human operator parameters associated with the humanoperator of the vehicle; determining an alertness level of the humanoperator based on the human operator parameters and the drivingcapability profile; receiving an indication to transfer control of thevehicle to the human operator; displaying a manual control requestnotification to the human operator, the manual control requestnotification communicating a request for manual control, the manualcontrol request notification further being based on the determinedalertness level; and transferring control of the vehicle to the humanoperator if the human operator accepts the request for manual control.2. The method of claim 1, further comprising: displaying a second manualcontrol request notification to the human operator if the human operatordoes not accept the request for manual control, the second manualrequest notification being different than the manual control requestnotification.
 3. The method of claim 1, further comprising: receiving ahazard indication of a road hazard near a route of the autonomousvehicle, and wherein the manual control notification is based at leastin part on the hazard indication.
 4. The method of claim 1, wherein thesensing operation includes facial analysis of the human operator.
 5. Themethod of claim 1, wherein the sensing operation includes a sensed heartrate of the human operator.
 6. The method of claim 1, wherein thedetermining operation includes body posture and body location of thehuman operator.
 7. The method of claim 1, wherein the displayingoperation depends at least in part on a baseline responsiveness level ofthe human operator, the baseline responsiveness level having beendetermined before the displaying operation.
 8. The method of claim 1,wherein accepting the request for manual control includes the humanoperator being physically in contact with a steering wheel.